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ABSTRACT 



In Kentucky, high stakes assessment of schools and school 
districts resulting from the state's educational reform initiative raises the 
issue of fairness. Should educational officials and teachers be held 
accountable for student performance despite vast differences in the 
socioeconomic conditions of the surrounding communities when extensive 
research shows community context affects student performance to a substantial 
degree? In this paper, a method is presented for assessing educational 
performance that corrects for community contextual factors. The issues of 
high stakes accountability and effective schools research are also discussed. 
The method relies on spatial analysis, a technique developed by geographers 
that entails mapping the standardized residuals from a regression analysis to 
find county school districts in which student performance is greater than 
expected, and less than expected, net of community context. Use of this 
procedure, it is argued, lessens possible bias in the Kentucky assessment 
results and helps allay concerns over unfairness in the state's 
accountability system. The procedure can be modified for a school -to-school 
analysis, although the mapping task would be more exacting in its smaller 
scale. Analysis of such spatial maps will allow educators to locate pockets 
of excellence and begin action research to identify particularly effective 
practices. An appendix discusses community level socioeconomic variables. 
(Contains 1 figure, 1 table, and 39 references.) (Author/SLD) 
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Abstract 

In Kentucky, high stakes assessment of schools and school districts resulting from the 
state's educational reform initiative raises the issue of fairness. Should educational officials and 
teachers be held accountable for student performance despite vast differences in the 
socioeconomic conditions of the surrounding communities when extensive research shows 
community context affects student performance to a substantial degree? In this paper, we present 
a method for assessing educational performance that corrects for community contextual factors. 
The issues of high stakes accountability and effective schools research are also discussed. 
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Spatial Analysis of Kentucky Assessment Results, 

Social and Economic Indicators 

In Kentucky, high stakes assessment of schools and school districts resulting from the 
state’s educational reform initiative raises the issue of fairness. Should educational officials and 
teachers be held accountable for student performance when the socioeconomic circumstances of 
school districts vary so much? Is not educational Achievement affected by the socioeconomic 
environment of the surrounding community? An extensive body of research buttresses this 
concern by showing that community socioeconomic context— factors over which local educators 
have little direct control— affect school climate and student performance to a substantial degree 
(Berliner & Biddle, 1995; Biddle, 1997; Bracey, 1997; Coleman, 1988, 1990; Coleman & Hoffer, 
1987; Crane, 1991; DeYoung, 1991; Garner & Raudenbush, 1991; Guskey andKifer, 1990; 
Hanushek, 1997; Lee, Smith & Croninger, 1997; Marino, 1995; Paul, 1996; Schneider & 

Coleman, 1993; Smith, Beaulieu & Israel 1992; Wenglinsky, 1997). 

In this paper we present a method for assessing educational performance that corrects for 
community contextual factors. We also demonstrate the use of spatial analysis, a technique 
crafred by geographers, showing how it can contribute to a better understanding of the Kentucky 
assessment results. The spatial analysis technique entails mapping the standardized residuals from 
a regression analysis to find county school districts in which student performance is greater than 
expected and less than expected, net of community context. Use of this procedure, we argue, will 
lessen possible bias in the Kentucky assessment results and thus will help to allay the concern over 
unfairness in the state’s accountability system. 

The approach of this paper spotlights performance differences among Kentucky school 
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districts, but we will also suggest how to compare the assessment results of individual schools 
using a simple modification of the method shown here. The technique also permits the assessment 
of improvement or decline in school and school district performance. 

High Stakes Assessment and Accountability 

Education reform is not new. The 1970s saw the basic skills movement and minimum 
competency testing. The 1980s moved fi-om expanded state mandates to reforming local school 
governance, as in site-based management (Cibulka & Berlin, 1995). A review of the literature on 
school improvement will identify the 1990s as the decade of school accountability (Sewall, 1996). 
Accountability is the generic term used to imply that teachers, administrators, schools, and school 
districts must answer to the public regarding the performance of their students (Bemauer & Cress, 
1997; Wolf, LeMahieu & Eresh, 1992). It is based on the proposition that if the state and federal 
governments are going to pump significant dollars into school districts they have a legitimate 
interest in seeing that the districts use these public funds to achieve the educational goals the state 
and federal governments want. Accountability is also based on the belief that once educational 
goals are clearly articulated, they can be measured with the ultimate goal of tying educational 
funding to the performance results (Cibulka & Berlin, 1995). Most often, accountability takes the 
form of assessments administered to students. It may also consist of assessment information 
combined with various performance measures such as attendance, dropout, and retention rates. 

There is no real consensus on the issue of efficacy of educational accountability programs. 
Most agree that the ultimate goal is the improvement of student learning. However, there is 
much debate regarding the type (standardized tests vs. performance measures) and setting (locally 
controlled vs. state developed and mandated) of the accountability assessments. Some educators 
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state that in order to be most effective, state and/or national assessments need to reflect the 
educational standards that the local community deems important (Bemauer & Cress, 1997; 
Lieberman & Miller, 1990). In fact, Theodore Sizer and his colleagues state that the proper locus 
for accountability is the local school (Sizer, McDonald & Rogers, 1993). Due to the premise that 
what is tested becomes what is taught (Resnick, 1993), these educators are now proposing that 
the best assessments reflect what the local community wants students to learn and are designed by 
local educators to demonstrate authentic learning (Smith, 1996; Wiggins, 1992). Additionally, 
they believe local educators must be personally invested in the standards and assessment if it is to 
effect any lasting change (Resnick, 1993). However, some educational reformers question the 
desirability of further standard setting and testing and debate both the efficacy and 
appropriateness of high stakes accountability testing (Cibulka & Derhn, 1995; Sizer, McDonald & 
Rogers, 1993). 

External assessments of accountability, in contrast, do not reflect locally agreed upon 
curricular goals or standards. Educators in favor of local placement of accountability efforts 
believe external assessments cause a stagnation of the curriculum by encouraging teachers and 
administrators to focus more time and resources on test-taking and less on what has been locally 
defined as important (Herman, 1992). Some educators argue for local development and local 
control while other educational reformers advocate the development and use of better batteries of 
tests to externally monitor schools and learning (Macpherson, 1995). In fact, some educators 
believe it is absolutely necessary to change the modes of assessment in order to change the 
schools (Smith, 1996). 

Despite concerns of appropriateness, accountability assessments are increasingly being 
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used as a means to facilitate school improvement (Cibulka & Berlin, 1995). Accountability 
measures are considered "high stakes" when significant rewards or sanctions result fi”om high or 
low student performance, respectively. The rewards or sanctions can be leveled at the students, 
resulting in promotion/graduation or retention, or the rewards can be leveled at the school or 
district in various ways. Schools or districts might be deregulated by having highly restrictive 
state or federal guidelines waived for outstanding student performance, or be compelled to submit 
laborious records of attempts to improve performance if students score below the acceptable 
level. A more recent approach to high stakes accountability involves rewarding and sanctioning 
teachers and administrators in schools dependent upon student performance. Pay for performance 
is based on successful private sector practices that reward team members for outstanding 
production (Kirst & Odden, 1993). An example of using accountability programs as a punitive 
means of control is reported by Smith (1996) who states that some vocal groups in Arizona 
decided the problem with their schools was lazy and incompetent teachers who needed to have 
their feet held to the fire through a state-mandated accountability assessment program. 

The Kentucky Education Reform Act (KERA) of 1990 implemented, among numerous 
and vast systemic educational changes, a primarily performance-based form of statewide student 
assessment; the Kentucky Instructional Results Information System (KIRIS). KIRIS was 
developed to provide the yardstick for determining whether the mandated KERA changes were 
eflfective. It consists of a combination of written examinations, portfolios, and performance 
events (which have now been omitted) administered to students at grades 4,5,7, 8, 1 1 and 12. It 
also includes school performance measures such attendance, dropout, and retention rates and the 
successful transition to adult life (indicated by enrollment in post-secondary training, the military 
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or employment) for graduating high school students. The KIRIS program is a very high stakes 
accountability system. Teachers in schools which surpass their expected school improvement 
scores are given significant monetary rewards. Schools which fail to reach their target scores are 
initially sanctioned through the mandatory development of educational improvement plans. 
Continued or severe lack of attainment of target performance scores leads to state assignment of a 
"distinguished educator" to lead the school improvement process and a resulting waiver of teacher 
and administrator tenure with possible job loss (Guskey, 1994). 

Effective Schools Research 

Educators are in the third decade of "effective schools" research to try to identify and 
clarify the within-school, within school district, and community demographic factors which affect 
student academic achievement. This research is commonly thought to have begun with the 
Equality of Educational Opportunity report which proposed that socioeconomic factors have a 
major impact on student achievement (Coleman, Campbell, Hobson, McPartland, Mood, & 
Weinfeld, 1966). Most of the research has explored factors within schools where student 
achievement is higher than would be predicted from the socioeconomic level of the school 
clientele and has been carried out in urban schools with high percentages of students from low 
income families (Creemers, Reynolds & Swint, 1996; Lezotte, 1994; Wimpleberg, Teddlie & 
Stringfield, 1989; Zigarelli, 1996). The effective schools literature is very sparse on the topic of 
how education in rural settings is impacted by the socioeconomic conditions of the community it 
serves. The present study, by looking at an entire state, focuses on both rural and urban 
education. Methodology already exists to identify pockets of excellence and pockets of poor 
academic performance controlled for the impact of socioeconomic factors. However, the 
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procedures and results are not "user friendly" for most public school educators. The mapping 
techniques identified in this study present a way for educators without strong research 
backgrounds and the general public to interpret test score distributions across a state. 

A Spatial Analysis of the Kentucky Assessment Results 

Kentucky has 120 county school districts and 56 independent school districts. The 
independent districts are enclaves within certain counties and typically center on a distinctive 
community. In the multiple regression analyses that follow we will first look at contextual effects 
on the independent districts combined with the county districts; then we will analyze each 
subgroup. The spatial analysis, however, will only be done for the county school districts because 
we cannot easily represent the independent school districts on a state map. 

" ariables 

The Kentucky Department of Education uses 2-year weighted mean KIRIS scores in 
determining school and district performance. The 2-year weighted mean smooths interannual 
fluctuations that can be caused by variations in student body composition. Thus, this summary 
statistic is a more stable indicator of student performance than the single-year mean. For the 
present study we used the 2-year weighted mean scores from 1992-93 and 1993-94 as the 
dependent variable. 

Our independent variables measure community socioeconomic context. We wanted to 
find local community factors that significantly correlate with the school district KIRIS score 
because such variables may indicate contextual influences on the KIRIS scores. When we correct 
for the effects of these community influences, the result will be a truer picture of how the school 
system itself supports student performance. 
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Using the National Center for Educational Statistics School District Data Book (CD- 
ROM) for Kentucky and information provided by the Kentucky Department of Education, we 
considered a variety of variables that were potentially correlated with the district KIRIS score (see 
Appendbc). In the end we selected two independent variables that in various multiple regression 
models were significant and substantively strong. These variables are: (1) median household 
income (in 1990) and (2) percent of high school graduates attending college (in 1991). Each 
variable represents major dimensions of community resources that can affect student performance. 
Median household income, for instance, is a measure of the financial resources within the school 
district. Yet it also probably measures other things. Income level captures consumer tastes that 
are locally prevalent, leisure-time activities, social and political awareness, and a broad array of 
other characteristics that closely associate with income. The percent of high school graduates 
attending college is related to income (for the independent and county district combined sample, r 
= .30; for the county district sample, r = .50). 'Yet it is also a distinct dimension in some contexts 
as we will see. This variable measures the total capacity of the local community for producing 
high school graduates who then sought a post-secondary education. Although this measure may 
also indicate the quality of the local school system, we propose that it is a broader indicator of the 
entire community’s educational support capacity. In addition, this variables gives a baseline 
indicator of this capacity in 1991, when KERA was first promulgated to the school districts. 
Analysis and Results 

The analysis goes forward in two stages. First, we will perform multiple regression 
analysis on the combined sample of independent and county school districts and then will do 
similar analyses on the independent and county school districts taken separately. Second, we will 




n .n 

‘j 



Spatial Analysis 10 



map the standardized residuals obtained from the regression model for the county districts. 

Multiple Regression 

Table 1 shows three multiple regression models. In the first model (the combined sample), 
we see that both median household income and percent high school graduates attending college 
are veiy significant and predict 46 percent of the KIRIS score variance. The model for the sample 
of independent districts has an even better fit, predicting 57 percent of the variance. As might be 
expected, the model for the county districts, since it analyzes a larger sample than the independent 
districts, predicts less variance (40 percent). Still, this is impressive. All these models confirm 
that commumty context substantially influences average student performance. The results shown 
in Model 3 are especially interesting. Median household income is strong and veiy significant in 
this model but high school graduates attending college is nonsignificant. For Kentucky county 
school districts the financial resources of the local population is such a key contextual factor that 
dominates college attendance and other factors. We tried, for example, entering other variables 
with median household income in the regression model for the county district sample. Income 
dominated these other variables as well.* But because college attendance was important in the 

other two regression models shown in Table 1, we have kept this variable in the analysis of the 
county district sample. 

We can get an idea of the magnitude of the contextual effects on the KIRIS scores by 
looking at the unstandardized regression coefficients in Table 1. The unstandardized coefficients 
in all three models show that a $10,000 increase in median family income increases the KIRIS 
score an average of more than 3 points. The effect is greatest for the county districts where the 
average increase reaches a high of 3.5 points. A 10 percent increase in college attendance is 
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associated with an average increase in the KIRIS score of just over 1 point for the combined 
sample and nearly 2 points for the independent districts but has no effect on the county district 
scores. Why this difference exists between the independent and county school districts is beyond 
the scope of the present paper but is an issue that warrants future investigation. However, it is 
likely related to the tendency for the independent districts to be either "highly desirable" schools 
which tend to draw the better students from the county school system or to be considered very 
undesirable schools with a high percentage of at-risk and impoverished students. 

Spatial Analysis ^ 

From the third model in Table 1 we obtained the standardized residuals and then 
categorized them into five levels. The levels capture the expected level of student performance by 
school district after we have subtracted the average effects of income and college attendance; 

Standardized residual Expected level of student performance 



-3.0 to -1.5 


Much worse than expected 


-1.5 to -.5 


Somewhat worse than expected 


-.5 to .5 


About as expected 


.5 to 1.5 


Somewhat better than expected 


1.5 to 3.0 


Much better than expected 



A more conservative breakdoAvn of categories would be; -3.0 to -2.0, -2.0 to -1.0, -1.0 to 1.0, 1.0 
to 2.0, 2.0 to 3.0. Using this stricter approach reduces the number of ‘much worse than 
expected’ school districts from eight to three. A similar reduction in the number of ‘much better 
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than expected’ school districts would also occur. Since the cut points for the categories are 
arbitrary, the final determination of the categories must await further experience with using the 
technique. Our purpose here is only to demonstrate the technique’s possibilities. 

The categorization scheme is used to show county school district performance on a map 
of Kentucky. The map reveals that when we subtract the effects of income and college 
attendance most of the county school districts in the central and western parts of the state are 
performing about as expected. Many school districts in the southeast, however, perform worse 
than expected. This is noteworthy. These school districts are located in the region having the 
lowest incomes and college attendance rates in the state. But since we have controlled for these 
contextual factors, the result suggests other deficiencies in this region. To draw conclusions 
about whether these deficiencies are intrinsic to the region’s school systems or are due to other, 
still unidentified community factors would be premature. Nevertheless, since the income variable 
appears to powerfully represent outside-school influences, it is hard to escape the suspicion that 
some deficiencies lie in the education process per se in these largely rural school districts. A 
second, smaller pocket of less than expected performance is found centered around Nicholas 
County. This is another largely rural area. But rural areas are not the only ones where student 
performances are less than expected. Three county school districts near Ashland, five districts in 
northern Kentucky near Cincinnati, and three districts near Louisville perform less well than 
expected. It may be that both ruralness as well as urbanicity are disadvantageous contexts as far 
as student achievement is concerned. The map hints at this conclusion but a more detailed study 
must be done before we can say this with confidence.'^ 

County school districts performing at better than expected levels are more evenly 
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distributed across the state than districts performing at less than expected levels. This suggests to 
us that the better than expected cases are more likely the result of unique circumstances within the 
school districts. With few exceptions (these being Lexington-Fayette County, Oldham County, 
and Greenup County), none of the better than expected school districts is found inside a 
metropolitan area and some are quite rural. Since the effects of income level and college 
attendance have been corrected, we suggest that the districts performing at better than expected 
levels had on average better prepared students and the strongest schools in 1992 to 1994. 

Discussion and Conclusion 

The procedure described above can be modified for a school-to-school analysis. This 
analysis is more difficult than the one we have shown using school districts because the contextual 
factors that impact a single school are hard to isolate. Nevertheless, the problem is not 
insurmountable. We might substitute the percent of students on free or reduced lunch program 
for median family income. This would give us a measure of the general income level of the 
families who send children to that school. To establish a baseline of the school’s educational 
capacity at the outset of KERA, we might use the percent of students promoted to the next grade 
level (or 12th graders graduating from high school). The multiple regression model would then 
be run with these independent variables. After obtaining the residuals, schools could grouped into 
categories as was done for the county districts. Schools performing better or worse than 
expected could then be identified. The school residuals could also be mapped. However, this 
mapping task would be more exacting than the county district mapping since a smaller scale map 
is required. Geographers regularly draw such maps using either street addresses or geographic 
coordinates to fix locations. The methodology is already worked out, and software exists to 
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expedite some of the more laborious steps. 

To assess improvements and declines in either school or school district scores involves 
another simple modification of the study used in this paper. KIRIS scores that are weighted 
means can be compared. The increase or decrease in a school’s or a district’s weighted mean 
scores is calculated as the simple difference between KIRIS scores at the later point and the 
earlier point in time. We then let this difference be the independent variable. The analysis is 
unchanged. Since the Kentucky Department of Education uses such differences in weighted mean 
scores to determine a school’s eligibility for rewards (or sanctions), we would like to see if the 
kind of analysis we have proposed here would change the assessment results. We hope to 
conduct such an analysis in the coming months. 

Further, by providing a way for educators to identify pockets of excellence, the effective 
schools research can expand into more rural and suburban areas. Analysis of the spatial map 
presented here will allow educators to locate pockets of excellence and begin action research 
projects to identify particularly effective practices for specific types of educational community 
settings. 

Assuming that states will want to continue, or even increase, accountability assessments of 
schools due to the large investment of state dollars the real issue is how to conduct assessments 
with externally imposed norms so that they are accurate and fair. The techniques employed to 
make the process fair must also be understandable by school personnel and the general public. 

The technique presented here represents both a fair and understandable analysis to help ensure 
that schools and districts are not held accountable for factors over which they have no control, 
e.g., parental income levels. By subtracting the effects of local context, we get a fairer and more 
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accurate assessment of how closely a school district approximates externally imposed standards. 
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Appendix 

Before doing the multiple regression analyses we explored a number of community level 
socioeconomic variables that are potentially associated with district KIRIS scores. 



Variable Correlation 

w/KIRIS scores 

Median household income .605 

Percent high school graduates attending college .516 

Average age of the population . 128 

Educational inequality within the population -.405 

Income inequality -.248 

Racial heterogeneity .046 

Total number of households -.013 



As shown, the KIRIS scores are more highly correlated with income and college attendance than 
with any other variables we have examined so far. 
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Guskey and Kifer (1990) obtained strikingly similar results using pre-KERA (1987) 
nomed baste skills test scores for Kentucky school districts. For some coiroborating 
evidence that uses KIRIS scores from schools in Lexington-Fayette County, see the 
Lejongton Herald-Leader feature article by Krista Paul (1996). 

We would like to thank Tim Pitts, Assistant Professor of Geography at Morehead State 
University, for showing us how to map residuals and Karen Helton, an undergraduate 
geography student, for producing the map. 

Itegression analyses done too late to be incorporated in this paper suggest that this is indeed 
the case. 
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Table 1. Unstandardized Regression Coefficients: Mean KIRIS Scores as a Function of Income 
and College Attendance* 



Independent 

variable 




Model 


% 

% 


1 

Independent & 
county districts 


2 

Independent 

only 


3 

County 

only 


Median HH income 


3.139E-04 


3.388E-04 


3.527E-04 




(9.236) 


(6.262) 


(8.283) 


% HS grad attending 


.107 


.181 


-2.246E-02 


college 


(5.537) 


(5.187) ., 


(-.912) 


Adj. R- 


.455 


.565 


.403 


N 


176 


56 


120 



*t-ratios in parenthesis. 
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